FarsTail: a Persian natural language inference dataset

Natural language inference (NLI) is known as one of the central tasks in natural processing (NLP) which encapsulates many fundamental aspects understanding. With considerable achievements data-hungry deep learning methods NLP tasks, a great amount effort has been devoted to develop more diverse datasets for different languages. In this paper, we present new dataset NLI task Persian language, also Farsi, dominant languages Middle East. This dataset, named FarsTail, includes 10,367 samples are provided both well indexed format be useful non-Persian researchers. The generated from 3,539 multiple-choice questions with least annotator interventions way similar SciTail dataset. A carefully designed multi-step process adopted ensure quality We results traditional and state-of-the-art on FarsTail including embedding such word2vec, fastText, ELMo, BERT, LASER, modeling approaches DecompAtt, ESIM, HBMP, ULMFiT provide solid baseline future research. best obtained test accuracy 83.38% shows that there big room improving current real-world applications investigate extent models exploit superficial clues, biases, partition set into easy hard subsets according success biased models. available at https://github.com/dml-qom/FarsTail